Add RegressionTest error message for when possible non-deterministic behavior is detected. #883
Conversation
…behaviour is detected.
|
Do you have an issue or ticket that initiated this work? |
Yes. I was working with @fbruton, who was using RegressionTest, and we could not initially figure out why RegressionTest was complaining about "differences" even though he was having RegressionTest generate the answer XML file. I looked into it and figured out that the place was producing "non-deterministic state". This means that each execution of the place with the same input IBDO produced different output IBDO's. The error message given by RegressionTest did not give any indication that this was the situation. This PR updates the error message so that the user of RegressionTest is given additional information on what is causing RegressionTest to show that there are "differences". |
src/test/java/emissary/test/core/junit5/RegressionTestUtil.java
Outdated
Show resolved
Hide resolved
This PR adds an error message when RegressionTest is generating answers and differences are detected. This usually indicates that there is non-deterministic state in the IBDO because the place is executed once to generate the answer file and once to check against the answer file. If these two executions produce different results, then this most likely indicates non-deterministic state.